Hierarchical Actor-Critic

The ability to learn at different resolutions in time may help overcome one of the main challenges in deep reinforcement learning — sample efficiency. Hierarchical agents that operate at different levels of temporal abstraction can learn tasks more quickly because they can divide the work of learning behaviors among multiple policies and can also explore the environment at a higher level. In this paper, we present a novel approach to hierarchical reinforcement learning called Hierarchical Actor-Critic (HAC) that enables agents to learn to break down problems involving continuous action spaces into simpler subproblems belonging to different time scales. HAC has two key advantages over most existing hierarchical learning methods: (i) the potential for faster learning as agents learn short policies at each level of the hierarchy and (ii) an end-to-end approach. We demonstrate that HAC significantly accelerates learning in a series of tasks that require behavior over a relatively long time horizon and involve sparse rewards.